CoreData : AUV Science Data Processing
This page last changed on Jun 03, 2021 by kgomes.

IMPORTANT: This document has been moved to the following location: https://docs.mbari.org/internal/se-ie-doc/systems/auvctd/auv-science-data-processing/. In order to update the document, you need to checkout the source material from this repository https://bitbucket.org/mbari/se-ie-doc/src/master/, edit the files and then check them back in. They will be deployed automatically

AUV Science Data Processing

This document describes the Processing steps of Autonomous Underwater Vehicle data within MBARI's Shore Side Data System. It assumes familiarity with general AUV mission execution procedures and the operation of CVS checkout of software, building code using Ant, executing Unix (or Windows) command-line tools and Matlab m-files.

Major sections of this document:

  • Step-by-step data processing instructions
  • Software checkout and repository code management
  • AUV Portal
  • Dorado Science Data Processing

Step-by-step data processing instructions

Receive notification of new Dorado mission data, usually an email indicating where the log files are located - most of the time the files are copied to mccann's Tempbox but sometimes the data are provided on a memory stick. The data are first copied to Tempbox where the following steps are executed (review the AUVCTD Processing Data Flow Diagram to familiarize yourself with the process):

  1. Log into elvis as ssdsadmin. Actually, you need to login with your own account, then users by issuing 'sudo -u ssdsadmin -i'
  2. Create tar ball of mission directory, e.g. (be sure to replace the example mission/survey file and directory names with the ones for the new data): 
    <101 elvis.shore.mbari.org /Tempbox/mccann> cd /mbari/Tempbox/mccann
<103 elvis.shore.mbari.org /Tempbox/mccann> tar cvf 2012.268.07.tar 2012.268.07
2012.268.07/
2012.268.07/SmartSamplerDataLog.log
2012.268.07/Gulper.log
...
2012.268.07/ctdDriver2.cfg
2012.268.07/gps.log
2012.268.07/m3dmgx1.log
<105 elvis.shore.mbari.org /Tempbox/mccann> gzip 2012.268.07.tar
  3. Copy tarball to /usr/local/auv_ctd and run the auvportal which submits the data to SSDS: 
    <255 elvis.shore.mbari.org /Tempbox/mccann> cp 2012.268.07.tar.gz /usr/local/auv_ctd
<256 elvis.shore.mbari.org /Tempbox/mccann> cd /usr/local/auv_ctd
<257 elvis.shore.mbari.org /local/auv_ctd> ./portal AUVCTD2012268 2012.268.07.tar.gz
Tue, 25 Sep 2012 09:27:43 PDT: processConnection() got message:
AUVCTD2012268;2012.268.07.tar.gz
dstDir.getName() = 2012268
Unpacking /usr/local/auv_ctd/2012.268.07.tar.gz
Copying /usr/local/auv_ctd/2012268/2012.268.07/ctdDriver2.log to /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07/ctdDriver2.log
Copying /usr/local/auv_ctd/2012268/2012.268.07/gps.log to /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07/gps.log
Copying /usr/local/auv_ctd/2012268/2012.268.07/isuslog.log to /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07/isuslog.log
Copying /usr/local/auv_ctd/2012268/2012.268.07/devices.cfg to /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07/devices.cfg
...
DEBUG [Thread-0] (XmlBuilder.java:538) - appendPrimitiveType called with method name isValidRole on object of class moos.ssds.metadata.DataProducer
DEBUG [Thread-0] (XmlBuilder.java:482) - appendElement called on class moos.ssds.metadata.DataProducer using method getClass
DEBUG [Thread-0] (XmlBuilder.java:278) - Converting java.lang.Class(class moos.ssds.metadata.DataProducer) to XML
Marshalled the AUV mission in /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07 into XML
Attempting to publish AUVCTD /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07 to SSDS server jnp://bob.shore.mbari.org:1099/

published XML describing mission AUVCTD /mbari/AUVCTD/missionlogs/2012/2012268/2012.268.07 to SSDS server jnp://bob.shore.mbari.org:1099/

If everything goes well in SSDS:Ingest/Ruminate an email message to auvctd@listserver.mbari.org will arrive shortly.
Tue, 25 Sep 2012 09:34:29 PDT: Done receiving AUV mission data.

    Note: With the move of the portal to from jeckel to elvis the mailing to auvctd@listserver.mbari.org has stopped working. This needs to be debugged and fixed.

  4. Process the LOPC data, e.g.: 
    <109 elvis.shore.mbari.org /u/ssdsadmin> cd ~/dev/auv_ctd/src/py/auv/
<109 elvis.shore.mbari.org /u/ssdsadmin> /bin/bash
<109 elvis.shore.mbari.org /u/ssdsadmin> export PATH="/usr/local/miniconda2/bin:$PATH"
<109 elvis.shore.mbari.org /u/ssdsadmin> source activate auv_ctd
(auv_ctd) bash-4.2$ processLOPC.pl 2013.224.02
(auv_ctd) bash-4.2$ exit
  5. Once the mission's instrument netcdf files have been created (check pismo:/opt/ssds/logs/ssds-updateBot.log for successful processing) the overall Dorado science data processing script can be edited with the new survey to be processed: 
    <102 elvis.shore.mbari.org /u/ssdsadmin> cd ~/dev/auv_ctd/src/matlab/doradosdp/process/
<103 elvis.shore.mbari.org /doradosdp/process> vi reprocess_surveys.m
  6. Add a line for the new survey to the end of the reprocess_surveys.m file, commenting out the previous survey that was processed: 
    % CANON September 2012 - laps around ESP drifter - last one for this campaign
%%AUVprocess_main('AUVCTD', 2012, ['258.00'],       server, ssdsLoadFlag, ssdsLoadsJar, localFlag, localDataDir, localCalDir, notifyEmail, skipMissionProc, [], 'ctd1')

% CANON September 2012 - laps around ESP drifter - last one for this campaign
AUVprocess_main('AUVCTD', 2012, ['268.07'],       server, ssdsLoadFlag, ssdsLoadsJar, localFlag, localDataDir, localCalDir, notifyEmail, skipMissionProc, [], 'ctd1')
  7. Execute the Matlab job with a shell script that properly sets the DISPLAY environment and executes reprocess_surveys.m with matlab: 
    <104 elvis.shore.mbari.org /doradosdp/process> RunReprocessJob
  8. Check for error messages. If everything works an email is sent to the notifyEmail setting in reprocess_surveys.m, usually auvctd@listserver.mbari.org.

Software checkout and repository code management

The Software for both the AUV Portal and the Science data processing are located in the auv_ctd module on the CVS server moonjelly.shore.mbari.org. To check the software to a system use your favorite CVS client, or on Unix use these commands:

cd <your_development_directory>
cvs -d :pserver:<login>@moonjelly:/home/cvs login
(enter cvs account password)
cvs -d :pserver:<login>@moonjelly:/home/cvs co -d auv_ctd -P auv_ctd
(about a minute wait for the files to be put in your directory)

(Replace <your_development_directory> with a subdirectory name of your choice and <login> with your moonjelly login name.)

If you have already done this software checkout step and want to update your copy of the software from the repository (which you would want to do if someone else on the team has commited some changes to the repository) then you can get these changes with the cvs update command, .e.g.:cd <your_development_directory>/auv_ctd
cvs update(Note that if new directories have been added to the repository you will need to repeat the above checkout command.)

If you have made changes in the software and wish to submit them back to the repository for others to use then use the cvs commit command, e.g.:cvs commit
(your default editor will open the files that have been changed for you to enter text describnig the changes you made)
 

AUV Portal

Build and start-up

The AUV Portal is the software that accepts data collected by the AUV. It constructs SSDS-compliant metadata, submits this mission metadata to SSDS and copies the data (.log and .cfg files) to the shore-side network storage location. After checking out the software from CVS, build it with Ant and run it (Note: you will need both Apache Ant and Java JDK 1.4 or higher on your system with the ANT_HOME and JAVA_HOME environment variables set).

Build it (ie. Type 'ant')
cd to the build/dist directory
type 'portal' (windows or UNIX):

> cd auv_ctd 
> ant (wait for a BUILD SUCCESSFUL message) 
> cd build\dist 
> portal 
> java \-cp .\activation.jar;.\auvctd.jar;.\ftp.jar;.\jbossall-client.jar;.\junit.jar;.\log4j.jar;.\siam.jar;
.\ssds.jar;.\tar.jar;.\xercesImpl.jar;.\xmlParserAPIs.jar;.\xom.jar;. moos.operations.portal.AUVPortalService
Wait for client...

AUV data submission to SSDS

To submit AUV mission data to SSDS, log into the vehicle computer and use the adx command to transfer the data to the AUV Portal that you have started with the above directions. The adx syntax is as follows:

usage: adx PortalName DataName login pass survey dir \[dir2 ...\]
PortalName is the hostname of the portal
DataName is the hostname where the auv data resides
login is the login account on DataName that the portal will use
pass is the password to the login on DataName
SSDS survey name (e.g., AUVCTD_2003_191)
dir is the FULL path name of the data directory to copy

For example, if auv-2-control is the name of the computer running
the Portal, mvc-dmo1 is the name of the AUV's computer, login1 and pass1 are the login name and password for the AUV computer, AUVCTD2004028 is the survey name, and /home/dorado1/auv/altex/onboard/logs/2004.028.05 is the location of the data files on the AUV then the command would be:

adx auv-2-control mvc-dmo1 login1 pass1 AUVCTD2004028 /home/dorado1/auv/altex/onboard/logs/2004.028.05

Here is a diagram representing the data flow from AUV through the Portal to
SSDS. (Option 2 is what is currently used by the auv_ctd Portal.)

 
Figure 1. Diagram of data flow between AUV, AUV Portal, and Shore Side Data System.

On the AUV computer (where you ran the adx command the screen will look something like this:

bash-2.00$ adx 134.89.13.105 mvc-dmo1 dorado1 dorado1 OCCO2003237 /home/dorado1/auv/altex/onboard/logs/2003.237.04
    /home/dorado1/auv/altex/onboard/logs/2003.237.04
; cd /home/dorado1/auv/altex/onboard/logs ...
; tar cvf 2003.237.04_data.tar 2003.237.04 ...

a 2003.237.04/abortMplan.cfg 1 Blocks
a 2003.237.04/bluefinBattlog.log 524 Blocks
a 2003.237.04/crossbow.log 35615 Blocks
a 2003.237.04/ctdDriver.cfg 2 Blocks
a 2003.237.04/ctdDriver.log 11179 Blocks
a 2003.237.04/ctdDriver2.cfg 2 Blocks
a 2003.237.04/ctdDriver2.log 11179 Blocks
a 2003.237.04/devices.cfg 4 Blocks
a 2003.237.04/dvl.cfg 7 Blocks
a 2003.237.04/dvl.log 5323 Blocks
a 2003.237.04/DvlData.bin 33617 Blocks
a 2003.237.04/dynamicControl.log 25404 Blocks
a 2003.237.04/events.log 0 Blocks
a 2003.237.04/gps.log 69 Blocks
a 2003.237.04/hydroscatlog.log 7627 Blocks
a 2003.237.04/isuslog.log 610 Blocks
a 2003.237.04/lisstlog.log 8626 Blocks
a 2003.237.04/missionfile.cfg.pp 11 Blocks
a 2003.237.04/navigation.log 31120 Blocks
a 2003.237.04/parosci.log 7632 Blocks
a 2003.237.04/syslog 11231 Blocks
a 2003.237.04/tailCone.log 7630 Blocks
a 2003.237.04/vehicle.cfg 7 Blocks
a 2003.237.04/workSite.cfg 4 Blocks
a 2003.237.04/2003.237.04_data.tar.gz 71527 Blocks
; gzip \-f 2003.237.04_data.tar ...
; mv 2003.237.04_data.tar.gz 2003.237.04/. ...
; Portal message: 134.89.32.26;dorado1;dorado1;OCCO2003237;/home/dorado1/auv/altex/onboard/logs/2003.237.04/2003.237.04_data
.tar.gz
bash-2.00$


On the portal computer the screen will display text something like this:

Wait for client...
accepted connection
processConnection() - end-of-stream
Wait for client...
processConnection() got message:
134.89.32.26;dorado1;dorado1;OCCO2003237;/home/dorado1/auv/altex/onboard/logs/2003.237.04/2003.237.04_data.tar.gz
Retrieve mission files from AUV host:
134.89.32.26: /home/dorado1/auv/altex/onboard/logs/2003.237.04/2003.237.04_data.tar.gz
Retrieving /home/dorado1/auv/altex/onboard/logs/2003.237.04/2003.237.04_data.tar.gz
All done
dstDir.getName() = OCCO2003237
Unpacking C:\dev\auv_ctd\build\dist\2003.237.04_data.tar.gz
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\2003.237.04_data.tar.gz
to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\2003.237.04_data.tar.gz
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\abortMplan.cfg to
\\Tornado\AUVCTD\OCCO2003237\2003.237.04\abortMplan.cfg \[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\bluefinBattlog.log
to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\bluefinBattlog.log \[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\crossbow.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\crossbow.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\ctdDriver.cfg to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\ctdDriver.cfg
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\ctdDriver.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\ctdDriver.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\ctdDriver2.cfg to
\\Tornado\AUVCTD\OCCO2003237\2003.237.04\ctdDriver2.cfg\[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\ctdDriver2.log to
\\Tornado\AUVCTD\OCCO2003237\2003.237.04\ctdDriver2.log\[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\devices.cfg to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\devices.cfg
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\dvl.cfg to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\dvl.cfg
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\dvl.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\dvl.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\DvlData.bin to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\DvlData.bin
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\dynamicControl.log
to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\dynamicControl.log \[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\events.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\events.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\gps.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\gps.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\hydroscatlog.log to
\\Tornado\AUVCTD\OCCO2003237\2003.237.04\hydroscatlog.log \[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\isuslog.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\isuslog.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\lisstlog.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\lisstlog.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\missionfile.cfg.pp
to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\missionfile.cfg.pp \[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\navigation.log to
\\Tornado\AUVCTD\OCCO2003237\2003.237.04\navigation.log\[WARNING\!\! Overwriting
existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\parosci.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\parosci.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\syslog to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\syslog
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\tailCone.log to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\tailCone.log
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\vehicle.cfg to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\vehicle.cfg
\[WARNING\!\! Overwriting existing file\]
Copying C:\dev\auv_ctd\build\dist\OCCO2003237\2003.237.04\workSite.cfg to \\Tornado\AUVCTD\OCCO2003237\2003.237.04\workSite.cfg
\[WARNING\!\! Overwriting existing file\] [http://ssds.shore.mbari.org/auvctd/OCCO2003237/2003.237.04] 
WARNING\!\! Device information is missing from \\Tornado\AUVCTD\OCCO2003237\2003.237.04\dynamicControl.log
WARNING\!\! Device information is missing from \\Tornado\AUVCTD\OCCO2003237\2003.237.04\events.log
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
moos.ssds.model.XmlBuilder: Attempted to call "public java.util.Collection
moos.ssds.model.Deployment.listChildDeployments(java.lang.String)", an
access method that requires an argument, SKIPPING
Marshalled the AUV mission in \\Tornado\AUVCTD\OCCO2003237\2003.237.04 into
XML
Attempting to publish \\Tornado\AUVCTD\OCCO2003237\2003.237.04 to SSDS.
SUCCESS\!\! published \\Tornado\AUVCTD\OCCO2003237\2003.237.04 to SSDS

Note: Detailed command-line instructions for "portaling" AUVCTD mission data into SSDS are available in the README file in the project or in the operational location on jeckel.shore.mbari.org in /usr/local/auv_ctd/ directory. There is also the portal.log file there which logs the output of the deamon process running on jeckel that receives and processes the data from AUV uploads from the vehicle.

The portal submission process creates self-described netCDF files of all the logged data from each instrument. To produce more useful data products the science data processing must be done; this is described in the next section.

Dorado Science Data Processing

Within the auv_ctd software module is the auv_ctd/src/matlab/process directory. This directory contains the m-files for processing the AUV instrument data such that they may be easily used for scientific investigation.

The products include:

  • "level-1" data = calibrated & aligned, in native (full) resolution. This is the basis for the quick-look section plots.
  • "level-2" data = level-1 decimated to limiting resolution (ISUS) so that we can serve data as [time, lat, lon, depth, variable 1:N] which is commonly needed.

A "byproduct" of the processing is the generation of processing metadata which is loaded into SSDS so that we can track data products and the software that produce them. Here is a graph snippet showing the level-1 and level-2 data products for an AUV survey

To produce these data products check out the auv_ctd software as described above to a system that has Matlab version 6 or higher. From the auv_ctd/src/matlab/doradosdp/process directory start Matlab and execute the AUVprocess_main script. The documentation for this script is:

> matlab \-nodesktop
< M A T L A B >
Copyright 1984-2002 The MathWorks, Inc.
Version 6.5.0.180913a Release 13                       Jun 18 2002
To get started, type one of these: helpwin, helpdesk, or demo.   
For product information, visit www.mathworks.com.  
>> help AUVprocess_main 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% AUVprocess_main.m: front end script to be modified by the AUV
% science data processing person or in an automated way by SSDS.
%
% $Id: ScienceDataProcessing.html,v 1.3 2004/05/19 23:17:52 mccann Exp $
%
% Usage:
%   AUVprocess_main \[ <campaign>, <year>, <start-year-day.mission>, <end-year-day.mission>, ..
%                     <ssds_load_flag>, <ssdsLoads_jar>, <local_flag>, <local_dir> \]
%
%   campaign:    Name of particular type of missions (e.g. 'AUVCTD', 'BIOLUM', 'OCCO')
%   year:        Year of the missions
%   start-year-day.mission:    First mission to use for the section (e.g. '338.15')
%   end-year-day.mission:      Last mission to use for the section (e.g. '338.18')
%   ssds_load_flag:            If 1 then load processing metadata into SSDS
%   ssdsLoads_jar:             Name of SSDS loads jar file, (e.g. ssdsLoads-prey.jar)
%   local_flag:                Set to 0 to use SSDS DODS urls for input data, 1 for local files
%   local_dir:                 If local_flag is 1 then use <local_dir> to read the input data
%
%
%
% Examples:
%   To process data from SSDS DODS-accessible datasets, loading SSDS metadata:
%     >> AUVprocess_main('BIOLUME', 2003, '338.15', '338.18', 1, 'ssdsLoads-prey.jar', 0, '')
%
%   To process data from local files, not loading SSDS metadata:
%     >> AUVprocess_main('BIOLUME', 2003, '338.15', '338.18', 0, '', 1, '/hosts/tornado/auvctd/')
%
%   Year-days must have leading zeros, i.e. they must be 3 digits.
%
%
% Files copied to web accessible locations by this function include:
%  - Matlab data file (.mat) of the constructed survey
%  - A png image file that is a summary of the data in the .mat file
%  - An ODV compliant spreadsheet file (.txt)
%  - A COARDS-compliant DODS accessible netcdf file (.nc) that contains the same data of the .mat file
%  - A log file (diary of this script)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%   
>>

So to process a set of missions into a survey execute with the start and end mission and the proper parameters for accessing the data and loading the metadata into SSDS. For example this command creates a survey from 3 missions using the DODS accessible netCDF files and loads the data into the SSDS server prey:

>> AUVprocess_main('BIOLUME', 2003, '245.02', '245.04', 1, 'ssdsLoads-prey.jar', 0, '')

Will produce a bunch of output to the screen, which is captured as a diary log file and saved in SSDS. When it is finished you should see the last few lines like:

Submitting metadata to SSDS...
done. ProcessRun id = 1802
Processing time: 289 seconds

Important note: If changes are made to any of the matlab m-files they should be checked back into the CVS repository before the AUVprocess_main script is run. This ensures that the ProcessRun Software and Resources will have the correct version number references in SSDS. It's easy to commit the changes. Just run 'cvs commit' from the Unix command line before running AUVprocess_main.

Two level 2 products produced by this processing step are a netCDF file of the aligned and ISUS time step sub-sampled variables and an Ocean Data View spreadsheet file. The ODV Spreadsheet file is not as efficient or as capable as a native ODV Collection (a .var file).

If one wishes to work with AUV survey data in ODV then all the files for a survey in \\tornado\AUVCTD\surveys\<year>\odv should be copied to a local directory so that one has write permission to the files.
Document generated by Confluence on Feb 04, 2026 08:22